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© Method and apparatus for spatially variant filtering. 

© A method for performing image processing is embodied in apparatus (10) that includes a classification 
circuit (86). The classification circuit (86) classifies each pixel (26) in an image partly on the basis of a 
comparison between (A) the variation among pixel values in a window (28) containing that pixel (26) and (B) an 

activity-level threshold (PREV MD) determined by a routine (Figure 4) that computes that threshold from other 

pixel values in the image. On the basis of this classification, a multiplexer (82) selects among the outputs of a 
low-pass circuit (70) and a high-pass circuit (74, 76, 78, and 80) to produce a replacement pixel value for the raw 
pixel value that a scanner (20) produces by scanning an image source (16). 
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DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS 

Figure 1 depicts a system 10 for recording in a memory 12 or displaying on a display device 14 the 
contents of an image source such as a microfilm 16. The display device 14 may be one of a number of 

5 types, such as a cathode-ray tube or the image-applying machinery in a photocopier. The microfilm might 
be illuminated by a light source 18 and the resulting image scanned by a scanner 20. The scanner 20 
includes circuitry that treats the resultant image as being divided into a plurality of pixels, and it generates 
signals representing digitized values of those pixels. This raw pixel-value output is then subjected to some 
type of image processing before being stored or displayed. 

w The image process may comprise a sequence of steps, or it may include only one. For the sake of 
example, Figure 1 shows two step-performing elements, one of which is a quantizer 22 for reducing the 
number of bits per pixel. Other embodiments of the invention may not include such a feature. In accordance 
with the present invention, however, all include an adaptive filter 24 whose effect varies in accordance with 
the image contents. In the illustrated embodiment, it alternately performs high-pass and low-pass filtering. 

75 To determine which, the filter classifies pixels in accordance with their relative values and the value 
variations in their neighborhoods, as will now be described. 

With each pixel in the image, such as the central pixel 26 of Figure 2, there are associated two 
windows. A larger window 28 is preferably sized to be about half the extent of the significant part of the 
imaging system's point-source function, i.e., of the light-intensity function that results from a single point 

20 source in the plane of the microfilm. The contents of this window are used to establish an average value M a 
ofthe values of all pixels within the larger window 28. This average can be a weighted average in which, for 
instance, the pixels closer to the central pixel 26 are weighted more heavily, but I prefer simply to use a 
straight average, in which all pixels are equally weighted. This value will be considered a threshold between 
those values that are considered background and those values that are considered to convey signal, i.e.. 

25 character, information, as will be explained in more detail below. 

To determine whether the pixel 26 is within the background or signal region, I compare the large- 
window neighborhood average M a with a local average M b computed over a smaller window 30. This 
window can be of any smaller size; in principle, in fact, it can be so small as to include only pixel 26. For 
noise-suppression purposes, however, I prefer to include a plurality of pixels. 

30 In addition to M a and M b , two further parameters are needed to classify a pixel. One of these is an 
"activity level" parameter M d , which is a measure of the variation in pixel value in a window around the pixel 
26 under consideration. For the sake of convenience, the illustrated embodiment employs for this purpose 
the same window 28 that is used to compute M a , but such a choice is not necessary. 

The variation may be computed in any number of ways; standard deviation, for instance, is a common 

35 measure of variation that could be used in accordance with the present invention. In the illustrated 
embodiment, however, I employ the absolute mean deviation. That is, for the illustrated embodiment, 



1 y+L-1 X+K-l 

40 M d = 2 2 |P„-Mj , (1) 

KL i=y j=x 

wnere p.. \ s the value of the jth pixel in the ith row, K and L are the width and height, respectively, of the 
45 window over which the variation is computed, M w is an average pixel value in that window (and is equal to 
M a in the illustrated embodiment), and the upper-left-corner pixel in that window is the xth pixel in the yth 
row. 

The determination of what filter operation to perform is then made by comparing the small-window 
average M b with the large-window average M a and comparing the activity level M d with an activity threshold 

so value MD THR, which is derived in a manner that will be described in more detail below. 

Figure 3 depicts the manner in which the illustrated embodiment uses these four parameters for 
classification. The abscissa is the smaller-window average M b , while the ordinate is the activity level M d . If 
M b is less than the product of some constant ki and the larger-window average M a , then the pixel is 
considered to be in a background region, and the output value is one computed in accordance with the 
55 following function. 

P 0 = P r Ai|M a -M b | ) (2) 
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where P c is the output pixel value, p, is the input pixel value, and A, is a constant This is the backaround 

Z° is t T aPP ' iCat |°: 0f eqU3ti0n (2) t6ndS ,0 drive the ^ into ^b^SSS? 

That ,s. n the case of the typical microfilm, where the lighter areas (which, in this case, have lower 
numencal values) represent the background, the light values are made even lighter 

thP VT ' npU V piX ®' ^ is 9 reater ,han a second constant k 2 times the larger-window average M a then 
the replacemen value depends on what the activity level M d is around that P ixel. If i, is greater fra'n the 
general activity level MD_THR, then the pixe. is rep.aced in accordance with the following equation 

Po=Pi + A2|M a -M b |, (3) 

where A, is a constant. I have typically used the same values for A, and A 2 , but this is not necessary 

Reflection reveals that the application of equations (2) and (3) has the effect of applying a hiqh-nass 

On the other hand, if a pixel whose filtered value M b exceeds k 2 M a is associated with a window whose 

relces 2 VF* ?" ^ T" M °- THR ' *" SyStem apP ' i6S 3 '-ct^ s * 

Imlh^ll h « T Xel W ' th Smal,er - window av ^°e M b and this acts to smooth the image and de- 

emphasize differences between pixel values. 

function ^ dePiCtS ^ T 9 ^ b6tWeen Mb = k,Ma 3nd Mb = k2M * as bei "9 sub * cted to the low-pass 
« P n Z : t 9 '° n ' W Ch smal, - window aver age is nearly equal to the large-window average no 
general var.at.on .s occumng, and any significant difference between the particular pixel value undS 

InTlSZT^ ° f SUrr ° Un din9 piX6,S Sh0U ' d be SUp P reSSed - * s a P-ctiL, maner I etk 

and k 2 be set by the system quant,zation error; that is, the region between M b = k, M a and M b = k 2 M a in the 

Srhw y ,T eSen,S the situation in which M b = M a to within the quantization error of the ca cu Ins 

MD THF \Z l a y TT S6Parate treatm8nt 3 h0rizontal ^ a "-difference region around M d = 
MD_THR I ke the vertical region around M b = M a , but I have found no particular advantage for such an 
arrangement ,n the application for which the illustrated embodiment is intended 

skilledtThP £ Tr zed that 4 the hi9h " and ,0W -P ass ^"^ons used here are merely exemplary. Those 
skilled in the art w.ll be aware of many others. On the right side of equations (2) and/or (3) for example P 
icouw be replaced with M b -or vice versa -and similar (although not identical) results wouid be obSned 

We now turn to the manner in which the activity threshold MD THR is computed in the illustrated 
embodiment. This va.ue is intended to approximate the general variation level (or, in so™ «J booCe 1 
Z J« 0r ™' tiP,e PerhapS With 3 C ° nStant added or subt ' acted > '"the image a TvMeZt 

! 6 J' T ^ t0 find SUCh 3 l6Vel W ° Uld require P asses throu 9 h imifl.. and there are 
speed as well as cost considerations in some applications that make a two-pass process undes ibT 
Accordingly the .Nustrated embodiment employs a routine iliustrated in Figure 4 for gradual aZShe 
activity threshold as the scanning proceeds. aajusimg tne 

thr Jnh^r 311 h PPr ° aCh in , the R9Ure 4 r ° Utine iS 10 k86p 3 Sin 9' e value of the activi *y ^reshold MD THR 

nSdil ? SC , an bUt t0 Upd3te " 31 the end of each scan ,ine accordance w¥h an 
ntermediate variat.on-.nd.cat.ng value IMG MD. The routine so varies IMG MD during the line scan that ! 

ITJ PReT MD^tThf th 7 ariati0nS J n ? Wind ° WS ° f ^ Pi * 6ls Wh ° Se window" v^^S 

« J? t>e ?!, nnin i 0 . f 6aCh ima9G ' the Fi9ure 4 routine ^itia'^es the activity threshold MD THR as well 
a . ^f^'f 65 ,MG - MD - PREV_MD, and BG MD used in the routine/as block 40 indicates Tn The 

o St h e tt° d ' men !' WWCh ***** reS0 ' Uti0n yie ' dS 8 Pixe, - value ™9 e of ' ha"" Sid a va ue 

oa^nf th. r " " ^ a " CritiCaL R9Ure 4 d6PartS fr ° m the USual f,ow - chart invention by depicting two 
otock 42 lTZ a :°r a T h Para,,el - P3rt ' WhiCh Wi " be d6SCribed below ' beains with d -isS 
PREV ur. Tin ?. , P ' Ch ,S emP '° yed t0 6Stab,ish the avera 9 e activi *V teve' (i-e.. variation) 

,™^ MD ; 'ow-frequency regions, begins with another decision block 44. Like MD THR, PREV MD is 

PREV MH ' a, H?H en<J ° f 6aCh SCan Mne - ' n interim ' 3 t6m P° rar V value BQ MD, froVwhLh 
PREV MD ,s updated, is adjusted by the part of the Figure 4 routine that starts with block"44 

The purpose of this block is to identify pixels that are most likely to be in low-spatial-frequencv reaions 

55 diffe I?:? 6 ' 44 emP '° yS 38 3 Criteri ° n ,he Q ^ 0f ( actua,, y> less t'an a quant zation^or 
55 difference between) M a and M b , i.e., between the larger- and smaller-window averages, .f the pfxel unde 
consideration meets this criterion, then BG MD is adjusted up or down slightly toward the Icl^ tTZ 

dtte:m ne1tra rt r G r Mr ,,S TL W r° W R 0 SpeCifiCal ^ if a ^ -Panted by deciS 'block 46 
determines that BG — MD exceeds M d , then BG_MD is decreased by 1/100 in a step represented by block 
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48; that is, step 48 must be reached one hundred times to change BG MD by one unit at the pixel-value 

resolution. If BG MD does not exceed M d , it is increased by 1/100 in a step represented by block 50. 

Such an adjustment of BG MD occurs for every pixel that meets the criterion of block 44. At the end 

of every scan line, the average low-frequency-region activity-level value PREV MD is updated to equal 

5 BG MD, as block 52 indicates. 

This PREV MD value is used in the first step, represented by block 42, of the other parallel routine. 

Specifically, the decision step of block 42 compares the current pixel's activity level M d with the low- 
frequency-region activity level PREV MD. If it is not at least equal to that low-frequency-region activity 

level, then the value of the variation within the current pixel's window will not be used to affect the activity 

io threshold MD THR. Otherwise, the temporary-value variable IMG MD from which the activity-level 

threshold MD THR is updated is adjusted slightly up or down toward the variation M d within the current 

pixel's window, as blocks 54, 56, and 60 indicate. Then, when the end of a scan line is reached as 

determined by a block-62 decision, the activity threshold MD THR is adjusted in accordance with the 

equation of block 64. As that block shows, the activity threshold MD THR is essentially adjusted half way 

75 from its previous value to the current value of IMG MD at the end of each line. (The "bias" represented by 

the " + 1 " in the equation is used only to overcome an artifact of the limited-precision arithmetic used in the 
calculation.) 

Figure 5 depicts a simplified block diagram of the type of hardware that might be employed for 
implementing this invention in real time. A line store 66 receives the raw pixel values from the scanning 

20 device and accumulates the pixel values in a group of scan lines equal in number to the height of the larger 
windows. At the pixel-scan rate, it applies to average-computing circuits 68 and 70 a window's worth of pixel 
values for the large and small windows, respectively. These circuits compute M a and M b , which they apply 
to a subtraction circuit 72. Circuit 72 sends the magnitude part of the result--i.e„ the absolute value-to two 
multipliers 74 and 76, which receive the high-pass-equation constants Ai and A 2 from appropriate registers 

25 not shown in the drawings. The values used for these constants are not critical, but I have satisfactorily 
employed a value of eight for both constants in a system in which the pixel values are given with eight-bit 
resolution. Adders 78 and 80 respectively subtract and add these values from and to the input pixel value 
for the pixel currently under consideration and apply the results to a three-input multiplexer 82. 

At the same time, a variation-calculating circuit 84 uses the output of the line store 66 and the M a value 

30 from averaging circuit 68 to compute the variation M d inthe larger window's pixel values. From this and the 

line-store output, it also generates the activity threshold MD THR. On the basis of the resulting MD THR 

and M d values and of an output of subtraction circuit 72 that indicates whether M b is greater than, equal to, 
or less than M a , a classifier 86 classifies the current pixel in accordance with the Figure-3 scheme and 
applies to the multiplexer 82 a select signal that represents the Figure-3 class to which the pixel currently 

35 under consideration belongs. The multiplexer 82 then selects among the (low-pass) M b value from the 
averaging circuit 70 and the two high-pass-filter outputs from adders 78 and 80 to produce the proper 
replacement pixel value. In the illustrated embodiment, the classifier 86 may additionally supply its output to 
the quantizer 22 if that quantizer is of the type described in my co-pending U.S. patent application for a 
Spatially Variant Filtering System filed on even date herewith. 

40 It is clear from the foregoing description that the broader teachings of the present invention can be 
employed in a variety of embodiments that differ significantly from that illustrated in the drawings. For 
instance, while the embodiment described in the drawings classifies pixels in accordance with two criteria, 
namely, intensity averages and variation averages, more or fewer criteria can be employed for the 
classification process. In particular, although the present invention is particularly advantageous for textual 

45 images, it can also be employed for other types of images, such as natural images, for some of which the 
background-versus-signal classification may not be helpful. For such images, one might use only the 
variation criterion or the variation criterion in connection with some other criterion. 

Also, the pixel data have been described as scalar values, which they are if the image is a gray-scale 
image or the classification is based on the luminance data from a color image. Clearly, however, the 

so broader aspects of the invention can be used on, for instance, the separate red, green, and blue 
components of an RGB color system. 

Additionally, even in systems that are very similar to that of the illustrated embodiment, many variations 
are possible. For instance, although the pixels are shown as being centered within the windows on which 
their processing is based, there may be applications for which some offset is desirable or convenient. 

55 Furthermore, although the same window is used in the illustrated embodiment for computing both the local 
activity level M d and the larger-neighborhood average M a , there is clearly no necessity that those windows 
be the same. Moreover, the same, smaller window is used both for distinguishing between background and 
signal regions and for performing low-pass filtering. Such an arrangement is not necessary, either. 
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Accordingly, the present invention can be used in a wide range of embodiments and thus constitutes a 
significant advance in the art. 

Claims 

1. In the method of processing an image of an image source that comprises scanning the image source to 
generate raw pixel values associated with respective pixels in a raw version of the source's image, 
generating processed pixel values by performing an image process on the raw pixel values, and storing 
the processed image values in an appropriate medium or displaying the image represented thereby 
the image process comprising a set of at least one processing step, each processing step in the 
sequence producing associated pixel-value outputs representing a version of the image by processing 
respective pixel-value inputs representing the values of corresponding pixels in at least one said 
version of the image, the improvement wherein one said processing step comprises: 

A) establishing a variation window associated with each pixel and computing the variation in said one 
processing step's input's pixel values associated with the pixels in that variation window; 

B) for each of a plurality of pixels in the image, making a process choice between performing and 
refraining from performing one of a high-pass and a low-pass operation thereon, the process choice 
depending on the variation computed for that pixel; and 

C) in accordance with the process choices, generating pixel output values by performing said one of 
a high-pass and a low-pass operation on some of the input pixel values but not performinq it on 
others. 

2. A method as defined in claim 1 wherein the variation employed in the step of making a process choice 
depends on whether the variation is above or below a variation threshold. 

3. A method as defined in claim 2 wherein the process choice depends on whether the variation is above 
or below a variation threshold. 

4. A method as defined in claim 3 wherein the variation threshold is variable and the variation-threshold 
value used to make the process choice for a given pixel depends on the variations in the variation 
windows of other pixels in the image. 

5. A method as defined in claim 4 wherein: 

A) the method further comprises: 

i) establishing a local-average window for each of at least some of the pixels in the image and 
computing a local average pixel value for that local-average window; and 

ii) establishing a neighborhood-average window for each pixel and computing a neighborhood- 
average pixel value for that local-average window; and 

B) the variation threshold used to make the process choice for any given pixel is a function only of 
pixels whose local averages differ from their neighborhood averages by less than a predetermined 
maximum. 

6. A method as defined in claim 1 wherein the process choice depends on whether the variation is above 
or below a variation threshold. 

7. A method as defined in claim 6 wherein the variation threshold is variable and the variation-threshold 
value used to make the process choice for a given pixel depends for any given pixel on the variations 
in the variation windows of other pixels in the image. 

50 8. A method as defined in claim 7 wherein: 

A) the method further comprises: 

i) establishing a local-average window for each of at least some of the pixels in the image and 
computing a local average pixel value for that local-average window; and 

ii) establishing a neighborhood-average window for each pixel and computing a neighborhood- 
average pixel value for that local-average window; and 

B) the variation threshold used to make the process choice for any given pixel is a function only of 
pixels whose local averages differ from their neighborhood averages by less than a predetermined 
maximum. 



30 



35 



40 



45 



55 



BNSDOCID: <EP 0531904A2J_> 



1 



EP 0 531 904 A2 



9. A method as defined in claim 1 wherein: 

A) the method further includes the steps of: 

i) establishing a local-average window for each of at least some of the pixels in the image and 
computing a local average pixel value for that local-average window; and 
5 ii) establishing a neighborhood-average window for each of at least some of the pixels in the 

image and computing a neighborhood-average pixel value for that local-average window; and 

B) the process choice further depends on whether the neighborhood average exceeds the local 
average. 

io 10. A method as defined in claim 9 wherein the process choice depends on whether the variation is above 
or below a variation threshold. 

11. A method as defined in claim 10 wherein the step of generating a pixel output value comprises: 

A) for each input pixel value for which the local average is on a first, background side of a first 
75 constant times the neighborhood average, performing a high-pass operation thereon to generate a 

corresponding output pixel value; and 

B) for each input pixel value for which the local average is on the other, signal side of a second 
constant times the neighborhood average, generating a corresponding output pixel value by: 

i) performing a high-pass operation on that input pixel if the variation in the pixel values in the 
20 variation window is greater than the variation threshold; and 

ii) performing a low-pass operation on that input pixel if the variation in the pixel values in the 
variation window is less than the variation threshold. 

12. In the method of recording an image of an image source that comprises scanning the image source to 
25 generate raw pixel values associated with respective pixels in the source's image, generating pro- 
cessed pixel values by performing an image process on the raw pixel values, and storing the process 
image values in an appropriate medium or displaying the image represented thereby, the image 
process comprising a sequence of at least one processing step, each processing step in the sequence 
producing associated pixel-value outputs by processing respective pixel-value inputs representing the 

30 values of corresponding pixels in an image, the input of the sequence's first processing step being the 
raw pixel values, and those of any other processing step being the pixel-value outputs of the previous 
step in the sequence, the improvement wherein one said processing step comprises: 

A) establishing a variation window associated with each pixel and computing the variation in said one 
processing step f s input's pixel values associated with the pixels in that variation window; 
35 B) establishing a local-average window for each of at least some of the pixels in the image and 

computing a local average pixel value for that local-average window; 

C) establishing a neighborhood-average window for each of at least some of the pixels in the image 
and computing a neighborhood-average pixel value for that local-average window; 

D) for each of a plurality of pixels in the image, making a process choice between performing and 
40 refraining from performing one of a high-pass and a low-pass operation thereon, the process choice 

depending on whether the neighborhood average exceeds the local average; and 

E) in accordance with the process choices, generating pixel output values by performing said one of 
a high-pass and a low-pass operation on some of the input pixel values but not performing it on 
others. 

45 

13. In an apparatus for processing an image of an image source that comprises means for scanning the 
image source to generate raw pixel values associated with respective pixels in a raw version of the 
source's image, means for generating processed pixel values by performing an image process on the 
raw pixel values, and means for storing the processed image values in an appropriate medium or 

so displaying the image represented thereby, the means for generating processed pixel values comprising 
a set of at least one processing circuit, each processing circuit in the set producing associated pixel- 
value outputs representing a version of the image by processing associated pixel-value inputs 
representing the values of corresponding pixels in at least one said version of the image, the 
improvement wherein one said processing circuit comprises: 
55 A) means for establishing a variation window associated with each pixel and computing the variation 

in said one processing circuit's input's pixel values associated with the pixels in that variation 
window; 
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17. An apparatus as defined in claim 16 wherein: 
A) the apparatus further comprises- 

* ^roirr.si 1 ;^ 3 " na " in ,te process **• » - - 

30 

35 20. A method as defined in claim 19 wherein: 
A) the apparatus further comprises- 



40 



45 21. An apparatus as defined in claim 13 wherein: 
A) the apparatus further includes: 

i) means for establishing a local-average window for each of at least 



50 



BMh^process choice further depends on whether the neighborhood average exceeds the ioca, 
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23. An apparatus as defined in Cairn 22 wherein the means for generating a pixel output va.ue comprises: 
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A) means for performing, for each input pixel value for which the local average is on a first, 
background side of a first constant times the neighborhood average, a high-pass operation thereon 
to generate a corresponding output pixel value; and 

B) means for generating, for each input pixel value for which the local average is on the other, signal 
side of a second constant times the neighborhood average, a corresponding output pixel value by: 

i) performing a high-pass operation on that input pixel if the variation in the pixel values in the 
variation window is greater than the variation threshold; and 

ii) performing a low-pass operation on that input pixel if the variation in the pixel values in the 
variation window is less than the variation threshold. 

In an apparatus for processing the image of an image source that comprises means for scanning the 
image source to generate raw pixel values associated with respective pixels in a raw version of the 
source's image, means for generating processed pixel values by performing an image process on the 
raw pixel values, and means for storing the process image values in an appropriate medium or 
displaying the image represented thereby, the means for generating processed pixel values comprising 
a set of at least one processing circuit, each processing circuit in the set producing associated pixel- 
value outputs representing a version of the image by processing associated pixel-value inputs 
representing the values of corresponding pixels in at least one said version of the image, the 
improvement wherein one said processing circuit comprises: 

A) means for establishing a variation window associated with each pixel and computing the variation 
in said one processing circuit's input's pixel values associated with the pixels in that variation 
window; 

B) means for establishing a local-average window for each of at least some of the pixels in the 
image and computing a local average pixel value for that local-average window; 

C) means for establishing a neighborhood-average window for each of at least some of the pixels in 
the image and computing a neighborhood-average pixel value for that local-average window; 

D) means for making, for each of a plurality of pixels in the image, a process choice between 
performing and refraining from performing one of a high-pass and a low-pass operation thereon, the 
process choice depending on whether the neighborhood average exceeds the local average; and 

E) means for generating pixel output values in accordance with the process choices by performing 
said one of a high-pass and a low-pass operation on some of the input pixel values but not 
performing it on others. 
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© Method and apparatus for spatially variant filtering. 



© A method for performing image processing is 
embodied in apparatus (10) that includes a clas- 
sification circuit (86). The classification circuit (86) 
classifies each pixel (26) in an image partly on the 
basis of a comparison between (A) the variation 
among pixel values in a window (28) containing that 
pixel (26) and (B) an activity-level threshold 
(PREV MD) determined by a routine (Figure 4) that 



computes that threshold from other pixel values in 
the image. On the basis of this classification, a 
multiplexer (82) selects among the outputs of a low- 
pass circuit (70) and a high-pass circuit (74, 76, 78, 
and 80) to produce a replacement pixel value for the 
raw pixel value that a scanner (20) produces by 
scanning an image source (16). 
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